MASKED LANGUAGE MODEL

Counterfactual Data Augmentation for Neural Machine Translation (2021.naacl-main)

Challenge:	euphemisms are ordinary-sounding words with a secret meaning that are used to conceal information . a primary motive of their use on social media is to evade content moderation efforts .
Approach:	They propose to use social media to detect euphemisms without human effort . they first perform phrase mining on a raw text corpus to extract quality phrases . then they use word embedding similarities to select a set of euphoristic phrase candidates .
Outcome:	The proposed algorithm shows 20-50% higher detection accuracies than baselines.

Copied to clipboard

Qi Liu, Matt Kusner, Phil Blunsom

FASPell: A Fast, Adaptable, Simple, Powerful Chinese Spell Checker Based On DAE-Decoder Paradigm (D19-55)

Challenge:	Neural machine translation models often rely on large-scale parallel corpora for training, exhibiting degraded performance on low-resource languages.
Approach:	They propose a method that interprets language models and phrasal alignment causally and generates augmented parallel translation corpora by sampling new source phrases from a masked language model.
Outcome:	The proposed method improves translation, backtranslation and translation robustness on IWSLT’15 English Vietnamese, WMT’17 English - German, and WMT'18 English – Turkish.

Copied to clipboard

Yuzhong Hong, Xianguo Yu, Neng He, Nan Liu, Junhui Liu

Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order (2020.acl-main)

Challenge:	Existing spell checkers for Chinese are based on denoising autoencoder and decoder paradigms that require a small amount of data to be effective.
Approach:	They propose a Chinese spell checker based on a new paradigm which consists of a denoising autoencoder and a decoder.
Outcome:	The proposed spell checker is faster, more Adaptable to simplified and traditional Chinese texts and has a much simpler structure to be as much Powerful in error detection and correction.

Copied to clipboard

Yi Liao, Xin Jiang, Qun Liu

Jointly Masked Sequence-to-Sequence Model for Non-Autoregressive Neural Machine Translation (2020.acl-main)

Challenge:	Large-scale pretrained language models such as masked language model (MLM) have brought significant improvements to many NLU and NLG tasks.
Approach:	They propose a probabilistic masking scheme for the masked language model and a model with a uniform prior distribution on the masking ratio.
Outcome:	The proposed model outperforms BERT on a bunch of downstream NLG tasks.

Copied to clipboard

Junliang Guo, Linli Xu, Enhong Chen

Contextual Text Denoising with Masked Language Model (D19-55)

Challenge:	masked language models have been used for natural language processing tasks but few studies have adopted it in the sequence-to-sequence models.
Approach:	They propose to combine encoder and decoder to train a masked sequence-to-sequence model . they propose to train the encoder more rigorously by masking the encoded input .
Outcome:	The proposed model achieves 27.69/32.24 BLEU scores on English-German/German-English tasks with 5+ times speed up compared with an autoregressive model.

Copied to clipboard

Yifu Sun, Haoming Jiang

Automatic Named Entity Obfuscation in Speech (2023.findings-acl)

Challenge:	Recent advances in NLP have been vulnerable to noisy inputs.
Approach:	They propose a contextual text denoising algorithm based on a ready-to-use masked language model that does not require retraining and can be integrated into any NLP system without additional training on paired cleaning training data.
Outcome:	The proposed algorithm can correct noise text and improve performance in several downstream tasks.

Copied to clipboard

Judita Preiss

Speech & Audio in NLP

Challenge:	Identifying, replacing and inserting replacement named entities synthesized using voice cloning into original audio reduces the likelihood of deanonymization.
Approach:	They propose to use a named entity recognition system built directly on speech to replace a masked language model and synthesize the replacement named entities using voice cloning.
Outcome:	The proposed system is built on the English speech NER dataset and tested on a sample of the LibriSpeech corpus.

BFClass: A Backdoor-free Text Classification Framework (2021.findings-emnlp)

Copied to clipboard

Zichao Li, Dheeraj Mekala, Chengyu Dong, Jingbo Shang

Robustness in NLP

Challenge:	Various trigger design strategies have been explored to attack text classifiers, however, defending such attacks remains an open problem.
Approach:	They propose a backdoor-free training framework that poisons a subset of training data by injecting trigger patterns and setting their labels as the target labels.
Outcome:	The proposed framework can detect all the triggers, remove 95% of poisoned training samples with very limited false alarms, and achieve almost the same performance as the models trained on benign training data.

Adversarial Sample Generation for Aspect based Sentiment Classification (2022.findings-aacl)

Copied to clipboard

null Mamta, Asif Ekbal

Sentiment Analysis

Challenge:	Existing approaches to attack adversarial samples in natural language processing are ineffective . initial attacks perturb characters or words in sentences, resulting in grammatical incorrect or out-of-context sentences.
Approach:	They propose an attack algorithm that generates adversarial samples for a given aspect, maintaining more semantic coherency.
Outcome:	The proposed method outperforms the state-of-the-art methods in perturbation ratio, success rate, and semantic coherence.

Prompt-based Logical Semantics Enhancement for Implicit Discourse Relation Recognition (2023.emnlp-main)

Copied to clipboard

Chenxu Wang, Ping Jian, Mu Huang

AnthroScore: A Computational Linguistic Measure of Anthropomorphism (2024.eacl-long)

Challenge:	Existing methods for identifying discourse relations without explicit connectives are limited by the availability of annotated data.
Approach:	They propose a method that injects knowledge relevant to discourse relation into pre-trained language models through prompt-based connective prediction.
Outcome:	The proposed method achieves outstanding performance against the current state-of-the-art models.

Copied to clipboard

Myra Cheng, Kristina Gligoric, Tiziano Piccardi, Dan Jurafsky

Distilling Word Meaning in Context from Pre-trained Language Models (2021.findings-emnlp)

Challenge:	Anthropomorphism is commonplace in people's interactions with technology . anthropomorphizing language can suggest undue accountability and agency in technologies .
Approach:	They propose an automatic metric of implicit anthropomorphism in language . they use a masked language model to quantify how non-human entities are implicitly framed as human by the surrounding context.
Outcome:	The proposed metric measures how non-human entities are implicitly framed as human by the surrounding context.

Copied to clipboard

Yuki Arase, Tomoyuki Kajiwara

Unsupervised Domain Adaptation for Sparse Retrieval by Filling Vocabulary and Word Frequency Gaps (2022.aacl-main)

Challenge:	Existing methods to transform contextualised representations weaken excessive effects of contextual information.
Approach:	They propose a self-supervised learning method that distils word meaning in context from a pre-trained masked language model.
Outcome:	The proposed method outperforms the state-of-the-art method for lexical semantics and STS estimation.

Copied to clipboard

Hiroki Iida, Naoaki Okazaki

NCPrompt: NSP-Based Prompt Learning and Contrastive Learning for Implicit Discourse Relation Recognition (2024.findings-emnlp)

Challenge:	IR models with a pretrained language model outperform lexical approaches like BM25 for vocabulary mismatch.
Approach:	They propose an unsupervised domain adaptation method by filling vocabulary gaps by expanding queries and documents through an MLM.
Outcome:	The proposed method outperforms the current state-of-the-art domain adaptation method on datasets with a large vocabulary gap from a source domain.

Copied to clipboard

Yuetong Rong, Yijun Mo

A Self-supervised Joint Training Framework for Document Reranking (2022.findings-naacl)

Challenge:	Recent prompt learning methods have demonstrated success in IDRR, but they fail to fully exploit critical semantic features shared among various forms of templates.
Approach:	They propose an NSP-based prompt learning and contrastive learning method for IDRR that transforms the IDRR task into a next sentence prediction task.
Outcome:	The proposed model can be used to classify the discourse relation sense between argument pairs without an explicit connective.

Copied to clipboard

Xiaozhi Zhu, Tianyong Hao, Sijie Cheng, Fu Lee Wang, Hai Liu

Text Smoothing: Enhance Various Data Augmentation Methods on Text Classification Tasks (2022.acl-short)

Challenge:	Pretrained language models have been successfully applied to a wide range of tasks . however, the pretraining tasks were based on the context of documents .
Approach:	They propose a self-supervised joint training framework with a method called Masked Query Prediction to establish semantic relations between given queries and positive documents.
Outcome:	The proposed framework outperforms existing models on document reranking tasks without further pre-training . it uses a self-supervised method to establish semantic relations between given queries and positive documents.

Copied to clipboard

Xing Wu, Chaochen Gao, Meng Lin, Liangjun Zang, Songlin Hu

Information Extraction & Text Mining

Challenge:	Experimental results show text smoothing outperforms data augmentation methods by a substantial margin.
Approach:	They propose to use a masked language model to convert a token to a smoothed representation by converting a sentence from its one-hot representation to 'controllable smoothes' they propose to combine text smoothing with other data augmentation methods to achieve better performance.
Outcome:	The proposed method outperforms mainstream data augmentation methods by a substantial margin on different datasets in a low-resource regime.

SODAPOP: Open-Ended Discovery of Social Biases in Social Commonsense Reasoning Models (2023.eacl-main)

Copied to clipboard

Haozhe An, Zongxia Li, Jieyu Zhao, Rachel Rudinger

Self-training Improves Pre-training for Few-shot Learning in Task-oriented Dialog Systems (2021.emnlp-main)

Challenge:	Existing diagnostic tests for detecting social biases in NLP models only detect stereotypic associations pre-specified by the designer.
Approach:	They propose an approach for automatic social bias discovery in social commonsense question-answering by substituting names associated with different demographic groups and generating many distractor answers from a masked language model.
Outcome:	The proposed approach uncovers model’s stereotypic associations between demographic groups and an open set of words.

Copied to clipboard

Fei Mi, Wanhao Zhou, Lingjing Kong, Fengyu Cai, Minlie Huang, Boi Faltings

Joint Energy-based Model Training for Better Calibrated Natural Language Understanding Models (2021.eacl-main)

Challenge:	Large-scale pre-trained language models have shown promising results for few-shot learning in task-oriented dialog (ToD) systems.
Approach:	They propose a self-training approach that iteratively labels the most confident unlabeled data to train a stronger Student model.
Outcome:	The proposed approach improves state-of-the-art pre-trained models in few-shot learning scenarios for task-oriented dialog (ToD) systems when only a small number of labeled data are available.

Copied to clipboard

Tianxing He, Bryan McCann, Caiming Xiong, Ehsan Hosseini-Asl

Graph-Based Phonetic Error Correction of Noisy ASR (2026.acl-industry)

Challenge:	Existing calibration methods rescale posterior distributions of classifiers after training.
Approach:	They propose to use a noise contrastive estimation technique to train an energy-based model during finetuning of pretrained text encoders.
Outcome:	The proposed model can reach a better calibration competitive to strong baselines with little or no loss in accuracy.

Copied to clipboard

Pratik Rakesh Singh, Mohammadi Zaki, Aneesh Mukkamla, Pankaj Wasnik

Multimodality

Challenge:	Automatic speech recognition systems produce residual transcription errors that affect semantically critical tokens.
Approach:	They propose a phonetic-based algorithm that combines phonetic graph modeling with contextual language understanding to improve automatic speech recognition.
Outcome:	The proposed framework decouples phonetic reasoning from contextual semantic selection and improves accuracy.

Segment, Mask, and Predict: Augmenting Chinese Word Segmentation with Self-Supervision (2021.emnlp-main)

Copied to clipboard

Mieradilijiang Maimaiti, Yang Liu, Yuanhang Zheng, Gang Chen, Kaiyu Huang, Ji Zhang, Huanbo Luan, Maosong Sun

Weakly Supervised Text Classification using Supervision Signals from a Language Model (2022.findings-naacl)

Challenge:	Recent state-of-the-art (SOTA) effective neural network methods have been used in Chinese word segmentation (CWS) However, the robustness of the previous neural methods is limited by the large-scale annotated corpus.
Approach:	They propose a self-supervised Chinese word segmentation approach with a straightforward and effective architecture.
Outcome:	The proposed approach outperforms previous methods on 9 different CWS datasets with single criterion training and multiple criteria training and achieves better robustness.

Copied to clipboard

Ziqian Zeng, Weimin Ni, Tianqing Fang, Xiang Li, Xinran Zhao, Yangqiu Song

Low-Resource NLP

Challenge:	Existing weakly supervised text classification methods require a large number of annotated data and human annotations are expensive.
Approach:	They propose to query a masked language model with cloze style prompts to obtain supervision signals.
Outcome:	The proposed method outperforms baseline methods on three datasets by 2%, 4%, and 3%.

Different Strokes for Different Folks: Investigating Appropriate Further Pre-training Approaches for Diverse Dialogue Tasks (2021.emnlp-main)

Copied to clipboard

Yao Qiu, Jinchao Zhang, Jie Zhou

Automatic Assistance for Academic Word Usage (2020.coling-main)

Challenge:	Pre-trained models can be fine-tuned on domain-specific unlabeled data . however, most further pre-training works just keep running the conventional pre- training task .
Approach:	They propose to add a further pre-training phase to the model to improve downstream tasks . they propose to use a domain-adaptive pre-tuning phase to fine-tune the models on unlabeled data .
Outcome:	The proposed method improves multiple task-oriented dialogue downstream tasks.

Copied to clipboard

Dariush Saberi, John Lee, Jonathan James Webster

Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little (2021.emnlp-main)

Challenge:	Experimental results show that lexical formality analysis helps improve the quality of substitution suggestions over a number of competitive baselines.
Approach:	They propose a system that automatically suggests lexical substitutions that aim to incorporate more academic vocabulary.
Outcome:	The proposed system improves the quality of substitution suggestions over a baseline that relies on the masked language model.

Copied to clipboard

Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, Douwe Kiela

HPT: Hierarchy-aware Prompt Tuning for Hierarchical Text Classification (2022.emnlp-main)

Challenge:	masked language models (MLMs) pre-train to model higher-order word co-occurrence statistics . authors suggest that such models have learned to represent syntactic structures prevalent in classical NLP pipelines . purely distributional information largely explains the success of pre-training, authors say .
Approach:	They propose to pre-train masked language models on sentences with random shuffled word order and show they still achieve high accuracy after fine-tuning on many downstream tasks.
Outcome:	The proposed model performs well according to parametric syntactic probes . the authors argue that the model is not all that different from earlier distributional models .

Copied to clipboard

Zihan Wang, Peiyi Wang, Tianyu Liu, Binghuai Lin, Yunbo Cao, Zhifang Sui, Houfeng Wang

uChecker: Masked Pretrained Language Models as Unsupervised Chinese Spelling Checkers (2022.coling-1)

Challenge:	Hierarchical text classification (HTC) is a multi-label classification problem with a complex label hierarchy.
Approach:	They propose a Hierarchy-aware Prompt Tuning method to handle HTC from a multi-label perspective using a dynamic virtual template and label words that take the form of soft prompts to fuse the label hierarchy knowledge.
Outcome:	The proposed method achieves state-of-the-art performance on 3 popular HTC datasets and is adept at handling imbalance and low resource situations.

Copied to clipboard

Piji Li

SpellBERT: A Lightweight Pretrained Model for Chinese Spelling Check (2021.emnlp-main)

Challenge:	Chinese Spelling Check (CSC) is a crucial and essential task in the area of natural language processing.
Approach:	They propose a framework to conduct unsupervised spelling error detection and correction using masked pretrained language models such as BERT.
Outcome:	The proposed model improves character-level and sentence-level accuracy, precision, recall, and F1-Measure on standard datasets.

Copied to clipboard

Tuo Ji, Hang Yan, Xipeng Qiu

Multimodality

Challenge:	Chinese Spelling Check is a nontrivial task because of the nature of ideographic language.
Approach:	They propose a pretrained model with graph-based extra features that captures erroneous patterns . they use a graph neural network to introduce radical and pinyin information as visual and phonetic features.
Outcome:	The proposed model can show competitive performance on OCR datasets where most errors are not covered by existing confusion set.

DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings (2022.naacl-main)

Copied to clipboard

Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang-Wen Li, Scott Yih, Yoon Kim, James Glass

Virtual Data Augmentation: A Robust and General Framework for Fine-tuning Pre-trained Models (2021.emnlp-main)

Challenge:	Recent work shows that finetuning pretrained models with contrastive learning makes it possible to learn good sentence embeddings without labeled data.
Approach:	They propose an unsupervised contrastive learning framework for learning sentence embeddings . they use a masked language model to mask out the edited sentence .
Outcome:	The proposed framework outperforms SimCSE on semantic textual similarity tasks by 2.3 absolute points.

Copied to clipboard

Kun Zhou, Wayne Xin Zhao, Sirui Wang, Fuzheng Zhang, Wei Wu, Ji-Rong Wen

DialogVED: A Pre-trained Latent Variable Encoder-Decoder Model for Dialog Response Generation (2022.acl-long)

Challenge:	Recent studies have shown that powerful pre-trained language models can be fooled by small perturbations or intentional attacks.
Approach:	They propose a framework for fine-tuning PLMs using a masked language model and Gaussian noise to augment semantically relevant examples with sufficient diversity.
Outcome:	The proposed framework improves the robustness of pre-trained language models and alleviates performance degradation under adversarial attacks.

Copied to clipboard

Wei Chen, Yeyun Gong, Song Wang, Bolun Yao, Weizhen Qi, Zhongyu Wei, Xiaowu Hu, Bartuer Zhou, Yi Mao, Weizhu Chen, Biao Cheng, Nan Duan

Dialogue Systems & Conversational Agents

Challenge:	Existing pre-trained dialog models shed light on various downstream tasks in natural language processing (NLP).
Approach:	They propose a dialog pre-training framework that introduces latent variables into the enhanced encoder-decoder pre-train framework to increase relevance and diversity of responses.
Outcome:	The proposed model achieves state-of-the-art on personaChat, DailyDialog, and DSTC7-AVSD datasets.

Readability Controllable Biomedical Document Summarization (2022.findings-emnlp)

Copied to clipboard

Zheheng Luo, Qianqian Xie, Sophia Ananiadou

Knowledge Enhanced Masked Language Model for Stance Detection (2021.naacl-main)

Challenge:	Existing controllable summarization systems for biomedical documents have little attention to readability control, leaving users with incompatible summaries .
Approach:	They propose a task of readability controllable summarization for biomedical documents to generate summaries that are incompatible with users' levels of expertise.
Outcome:	The proposed model is based on pre-trained language models with prevalent controlling and generation techniques and evaluates the readability discrepancy between lay and technical summaries.

Copied to clipboard

Kornraphop Kawintiranon, Lisa Singh

Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction (2020.acl-main)

Challenge:	Detecting stance on Twitter is difficult because of the short length of each tweet . Twitter content is dynamic, constantly coining new terminology and hashtags .
Approach:	They propose a BERT-based fine-tuning method that enhances stance detection models . they use weighted log-odds-ratio to identify words with high stance distinguishability .
Outcome:	The proposed method outperforms the state-of-the-art for stance detection on Twitter data about the 2020 US presidential election.

Copied to clipboard

Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, Kentaro Inui

Paragraph-level Simplification of Medical Texts (2021.naacl-main)

Challenge:	Existing methods for incorporating a masked language model into an EncDec model have potential drawbacks when applied to GEC.
Approach:	They propose to incorporate a pre-trained masked language model (MLM) into an encoder-decoder model for grammatical error correction.
Outcome:	The proposed method achieves state-of-the-art on BEA-2019 and CoNLL-2014 benchmarks.

Copied to clipboard

Ashwin Devaraj, Iain Marshall, Byron Wallace, Junyi Jessy Li

Text Generation

Challenge:	Existing methods for simplification of medical texts are limited due to jargon and technical content.
Approach:	They propose to automate the simplification of medical texts by penalizing decoders for producing "jargon" terms.
Outcome:	The proposed method improves on existing heuristics by penalizing the decoder for producing "jargon" terms.

Contextualized Perturbation for Textual Adversarial Attack (2021.naacl-main)

Copied to clipboard

Dianqi Li, Yizhe Zhang, Hao Peng, Liqun Chen, Chris Brockett, Ming-Ting Sun, Bill Dolan

Robustness in NLP

Challenge:	Existing techniques for generating adversarial examples are driven by local heuristic rules that are agnostic to the context, resulting in unnatural and ungrammatical outputs.
Approach:	They propose a ContextuaLized AdversaRial Example generation model that generates fluent and grammatical outputs through a mask-then-infill procedure.
Outcome:	The proposed model outperforms baseline models in terms of attack success rate, textual similarity, fluency and grammaticality.

Probing Simile Knowledge from Pre-trained Language Models (2022.acl-long)

Copied to clipboard

Weijie Chen, Yongzhu Chang, Rongsheng Zhang, Jiashu Pu, Guandan Chen, Le Zhang, Yadong Xi, Yijiang Chen, Chang Su

Cross-Domain Review Generation for Aspect-Based Sentiment Analysis (2021.findings-acl)

Challenge:	Existing approaches to learn generic knowledge from a large corpus are time-consuming and labor-intensive.
Approach:	They propose a framework to probe simile knowledge from pre-trained language models to solve SI and SG tasks.
Outcome:	The proposed framework solves the SI and SG tasks in a simile triple completion task.

Copied to clipboard

Jianfei Yu, Chenggong Gong, Rui Xia

Sentiment Analysis

Challenge:	Existing domain adaptation methods for Aspect-Based Sentiment Analysis lack finegrained labeled data.
Approach:	They propose a new domain adaptation paradigm called cross-domain review generation which aims to generate target-domain reviews with fine-grained annotation based on the labeled source domain.
Outcome:	The proposed approach is superior to state-of-the-art domain adaptation methods.

Frustratingly Easy Edit-based Linguistic Steganography with a Masked Language Model (2021.naacl-main)

Copied to clipboard

Honai Ueoka, Yugo Murawaki, Sadao Kurohashi

Comparing Text Representations: A Theory-Driven Approach (2021.emnlp-main)

Challenge:	linguistic steganography is the practice of concealing a secret message in some cover data such that an eavesdropper is not even aware of the existence of the secret message.
Approach:	They propose to use edit-based linguistic steganography to generate genuine-looking texts by using a masked language model that eliminates painstaking rule construction and has a high payload capacity.
Outcome:	The proposed method eliminates painstaking rule construction and has a high payload capacity for an edit-based model.

Copied to clipboard

Gregory Yauney, David Mimno

Text Classification

Challenge:	Recent advances in NLP have been made by learning representations that transform complex tasks into simple classification tasks.
Approach:	They propose a method to evaluate the compatibility between representations and tasks by fitting text features to specific characteristics of text datasets.
Outcome:	The proposed model provides a calibrated, quantitative measure of the difficulty of a classification-based NLP task.

XLM-D: Decorate Cross-lingual Pre-training Model as Non-Autoregressive Neural Machine Translation (2022.emnlp-main)

Copied to clipboard

Yong Wang, Shilin He, Guanhua Chen, Yun Chen, Daxin Jiang

Multilinguality

Challenge:	Existing pre-training language models have been successful in natural language understanding and autoregressive generation tasks, but non-autoregressive models have not been sufficiently successful.
Approach:	They propose a pre-trained masked language model (MLM) and a non-autoregressive generation model with a lightweight decorator.
Outcome:	The proposed model outperforms the previous mask-predict model on translation datasets by 19.9x.

LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention (2020.emnlp-main)

Copied to clipboard

Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, Yuji Matsumoto

Detecting Erroneously Recognized Handwritten Byzantine Text (2023.findings-emnlp)

Challenge:	Existing models for entity representations do not capture information in a knowledge base, and cannot represent entities that do not exist in the KB.
Approach:	They propose a pretrained contextualized representation of words and entities based on the bidirectional transformer.
Outcome:	The proposed model achieves impressive empirical performance on a wide range of entity-related tasks.

Copied to clipboard

John Pavlopoulos, Vasiliki Kougia, Paraskevi Platanou, Holger Essler

Text Classification

Challenge:	Handwritten text recognition (HTR) produces textual output that contains errors, which are much higher than recognised printed text.
Approach:	They investigate the properties of handwritten texts that lead post-correction systems to this adversarial behaviour in Byzantine Greek.
Outcome:	The proposed model achieves an average precision score of 95% in Byzantine Greek and 97% in modern and ancient Greek.

SlovakBERT: Slovak Masked Language Model (2022.findings-emnlp)

Copied to clipboard

Matúš Pikuliak, Štefan Grivalský, Martin Konôpka, Miroslav Blšták, Martin Tamajka, Viktor Bachratý, Marian Simko, Pavol Balážik, Michal Trnka, Filip Uhlárik

What to Fuse and How to Fuse: Exploring Emotion and Personality Fusion Strategies for Explainable Mental Disorder Detection (2023.findings-acl)

Challenge:	SlovakBERT is a new masked language model that is based on a Web-crawled corpus.
Approach:	They introduce a new Slovak-only transformers-based language model called SlovkBERT . they evaluate the model on several NLP tasks and establish a benchmark for Slovakia .
Outcome:	The proposed model achieves state-of-the-art on several NLP tasks and achieves best results . the proposed model could be used by other Slovak researchers or NLP practitioners .

Copied to clipboard

Sourabh Zanwar, Xiaofei Li, Daniel Wiechmann, Yu Qiao, Elma Kerz

FACTIFY-5WQA: 5W Aspect-based Fact Verification through Question Answering (2023.acl-long)

Challenge:	Mental health disorders (MHD) are one of the greatest challenges facing our healthcare systems and modern societies in general.
Approach:	They integrate and extend the research by conducting extensive experiments with three types of deep learning-based fusion strategies: feature-level fusion, model fusion and task fusion.
Outcome:	The proposed techniques show that they can be used to improve mental health detection from textual data.

Copied to clipboard

Anku Rani, S.M Towhidul Islam Tonmoy, Dwip Dalal, Shreya Gautam, Megha Chakraborty, Aman Chadha, Amit Sheth, Amitava Das

Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning (2020.emnlp-main)

Challenge:	Contemporary fact-checking systems focus on estimating truthfulness using numerical scores which are not human-interpretable.
Approach:	They propose a 5W framework for question-answer-based fact explainability that can assist human fact-checkers in asking relevant questions . they propose masked language model which generates QA pairs for claims and a baseline QA system that automatically locates those answers from evidence documents.
Outcome:	The proposed framework can assist human fact-checkers in asking relevant questions related to a fact, which can then be validated separately to reach a final verdict.

Copied to clipboard

Tao Shen, Yi Mao, Pengcheng He, Guodong Long, Adam Trischler, Weizhu Chen

Exploring and Verbalizing Academic Ideas by Concept Co-occurrence (2023.acl-long)

Challenge:	Existing methods for integrating knowledge graphs into pre-trained language models have been poorly implemented.
Approach:	They propose a self-supervised entity masking scheme that exploits relational knowledge underlying the text.
Outcome:	The proposed model achieves improved performance on five benchmarks, including question answering and knowledge base completion.

Copied to clipboard

Yi Xu, Shuqian Sheng, Bo Xue, Luoyi Fu, Xinbing Wang, Chenghu Zhou

How to Train BERT with an Academic Budget (2021.emnlp-main)

Challenge:	a new framework for academic idea inspiration is being developed for academic research assistants . number of academic publications is increasing exponentially, making it difficult for an independent researcher to understand these papers thoroughly.
Approach:	They propose a framework based on concept co-occurrence for academic idea inspiration . they construct evolving concept graphs according to the co-existence relationship of concepts from 20 disciplines or topics .
Outcome:	The proposed system can be used to explore connections between academic concepts and verbalize the new ideas.

Copied to clipboard

Peter Izsak, Moshe Berchansky, Omer Levy

Towards Unsupervised Recognition of Token-level Semantic Differences in Related Documents (2023.emnlp-main)

Challenge:	Large language models such as BERT are used in many NLP tasks, but their pretraining phase can be prohibitively expensive for startups and academic research groups.
Approach:	They propose a recipe for pretraining a large language model in 24 hours using a low-end deep learning server.
Outcome:	The proposed model can be trained on GLUE tasks at fraction of the cost of pretraining.

Copied to clipboard

Jannis Vamvas, Rico Sennrich